pre-alpha: Season Pass / Wishlist / Logo state backup & restore tool

angra · 01-29-2003, 11:20 AM

The volume of people mentioning a need of such a thing has motivated me to get off my duff enough to package the tool I had made. This tools is intended for making backups, from running tivos, of important state information. The tool backs up and restores:

season passes
wishlists
logo assignments

This software is AS-IS with no warranty. It will probably make your tivo turn into a doorstop. Please don't run this (especially the restore!) if you are not prepared for your tivo drive to be rendered useless. This means backups, being prepared to make new backups and restore from backup, etc.

That said, I'm interested in hearing feedback from anyone who runs this. I don't have time to maintain this, so I can't promise timely fixes. I will not be offended at all if other people decide to take this code and "fork" it, so to speak, if I don't respond to requests quickly enough to suit.

to use, unpack and read the README.

you can PM me here or mail me at danspam@envy.gtf.org.

good luck and enjoy.

(edited 1/29/03 to clarify description)

Note below that AGW has posted a modified version with important corrections.

(eidted 2/24/03 to point to AGW bugfixed release)

zaknafein · 01-29-2003, 12:49 PM

Hey, nice. I assume this is geared for online backup?

angra · 01-29-2003, 12:51 PM

Yes, sorry I was not explicit about this.

This script is intended for an operating Tivo.

mrtickle · 01-29-2003, 06:21 PM

Thanks!

The readme says that if you do a backup and restore without deleting wishlists/sps, you end up with two of everything. But what about logos, should you delete them too or will the restore overwrite existing ones? I see it's easy enough to comment out the "restore_logos" line so I'll probably do that.

Oh one other thing, which software version did you use it with? Hopefully 2.5 and above, we have 2.5.5 here.

TIA

angra · 01-29-2003, 06:25 PM

Well I'm currently on 3.0.

I _think_ I did some of this with 2.5, but I don't remember for sure. I am reasonably sure that nothing major changed in the SP/wishlist structure between those versions though.

The logo stuff won't duplicate - it sets values in records that already exist, whereas the SP/WL stuff creates new records.

SteakMan · 02-04-2003, 02:48 PM

Has anyone tried it on a DTiVo?

I'll make a backup tonight, but I don't know when I'll have time to try a restore.

So, since this essentially re-creates entries instead of restoring, it might be adapted to adjust your SP lineup offline

-SteakMan-

angra · 02-04-2003, 02:51 PM

yes, that could definitely work.

if you feel adventurous, the backup file is pretty easy to read and you could probably tweak it.

It doesnt completely automate the process though - you'd still have to delete your SPs on the tivo first. You could modify the restore script to delete existing SPs first though if you were so inclined.

mrtickle · 02-04-2003, 06:18 PM

Quote:

Originally posted by angra
Well I'm currently on 3.0.

I _think_ I did some of this with 2.5, but I don't remember for sure. I am reasonably sure that nothing major changed in the SP/wishlist structure between those versions though.

The logo stuff won't duplicate - it sets values in records that already exist, whereas the SP/WL stuff creates new records.

Righto - thanks!

agw · 02-21-2003, 01:53 PM

I've just tried this script on my UK TiVo running 2.5.5 and unfortunately, while it picked up the wishlists and logos no problem, it didn't pick up the season passes.

This is the entire restore_seasonpasses proc that it writes:

proc restore_seasonpasses {} {
global new_tms
global new_tms_fs
global newids
global oldids
#
# SEASON PASSES
#
set new_tms ""
set new_tms_fs ""
}

I've had a little play with the script and changing the line that does the ForeachMfsFile for the season passes to use the directory "/SeasonPass" instead of "/SeasonPass/User" appears to do the trick and fill in the season passes.

This directory name corresponds with the directory name for the season passes in TiVoWeb.

I've just bought a couple of 120Gb drives so I'll let you know if the script loads the season passes back on after the upgrade :-) It'll be SO cool if it does. The thought of putting those back in by hand gives me the shivers... especially having to redo the priorities. It's a shame TiVoWeb doesn't give you an interface for setting the priorities because it's a nightmare doing it on the TiVo... too sloooooooooooooooow...

agw · 02-21-2003, 01:59 PM

Just to clarify before everyone points out that MFSTools will carry season passes over... this TiVo has already been upgraded once and MFSTools refuses to produce anything smaller than a 160 hour backup for it (I've used the -s option, no difference - ditto the -v(? can't remember) option that looks at stream sizes and not file sizes).

So I've taken a backup of a friends 40 hour 2.5.5 (I have my original discs but they're ancient) and I'll be restoring THAT onto the 120 Gb discs and expanding it.

Hence the reason why a season pass backup and restore script will be so useful.

angra · 02-21-2003, 02:06 PM

sounds great! I'm glad you were able to find and correct the problem.

I'll try to put together a modified version that reacts properly to UK tivos.

jodell · 02-21-2003, 08:25 PM

angra,

Any chance exporting "thumbs" could be added? I just added tivo #2 to my household and I want to move the new tivo to the living room. I will never get it done if i have to manually restore all the SP, wishlists and thumbs.

Thanks,
Jeff

angra · 02-21-2003, 09:30 PM

hi, I tried to do thumbs, and I could reliably restore the thumbs themselves but it never seemed to impact the suggestions engine.

The code is in the utilities as posted, just commented out, but as I said it doesn't quite work. Shows, actors, etc, will all be thumbed appropriately, but the prediction engine won't use this info properly.

mrtickle · 02-23-2003, 04:12 AM

Quote:

Originally posted by agw

I've had a little play with the script and changing the line that does the ForeachMfsFile for the season passes to use the directory "/SeasonPass" instead of "/SeasonPass/User" appears to do the trick and fill in the season passes.

This directory name corresponds with the directory name for the season passes in TiVoWeb.

Interesting, thanks for that. I think the /User structure was from software 3.0 onwards, which we don't have of course.

Quote:

I've just bought a couple of 120Gb drives so I'll let you know if the script loads the season passes back on after the upgrade :-) It'll be SO cool if it does. The thought of putting those back in by hand gives me the shivers... especially having to redo the priorities. It's a shame TiVoWeb doesn't give you an interface for setting the priorities because it's a nightmare doing it on the TiVo... too sloooooooooooooooow...

Putting mine back by hand I wouldn't mind at all. What would annoy me is losing all my Hibernating Season Passes, I have loads of these which are not currently airing and which CANNOT be recreated, unless you use angra's wonderful script! I definitely do NOT want to use wishlists for these! The last time I thought it would be a "good idea" to use wishlists (big mistake) it took me a full year to get the SPs back.

LJ · 02-23-2003, 07:57 AM

Quote:

Originally posted by angra
hi, I tried to do thumbs, and I could reliably restore the thumbs themselves but it never seemed to impact the suggestions engine.

How long did you wait after importing to check for new suggestions? Looking in some of the test scripts in \tvlib\tcl\tv it looks like TiVo will take at least a few minutes to respond to a change in thumbs.

Also I wonder if there's an event that needs to be sent to make TiVo see a change in thumbs if the change is made from a script rather than using the UI? For example ui.itcl sends a 'data changed' event when a season pass is updated - maybe a change in thumbs has it's own 'data changed' event?

angra · 02-23-2003, 09:36 AM

I waited several hours - I think the last test I ran I waited about 12. It was long enough to populate the suggestions list, but it was filling it up with stuff on the restored system that bared no resemblance to the original one.

you may be (probably are) right about the special event that needs to be sent.

i went from having time to work on this stuff to not, about 4-6 months ago. I had some good tivoweb mods 3/4 finished too :/. Anyway I just dont have time to continue working on it seriously, especially since testing the thumbs engine stuff is very disruptive to my tivoing.

StanSimmons · 02-23-2003, 11:12 AM

Quote:

Originally posted by agw
I've just tried this script on my UK TiVo running 2.5.5 and unfortunately, while it picked up the wishlists and logos no problem, it didn't pick up the season passes.

This is the entire restore_seasonpasses proc that it writes:

proc restore_seasonpasses {} {
global new_tms
global new_tms_fs
global newids
global oldids
#
# SEASON PASSES
#
set new_tms ""
set new_tms_fs ""
}

The same is true for US DTiVo's still running v2.5.

agw · 02-23-2003, 08:58 PM

Well, I ran the restore script and unfortunately it failed. The first part of restore restores the logos, which calls the file generated by backup, which in turn calls set_station_logo, which takes the tmsid it gets passed and tries to translate this to an fsid with station_tms_to_fsid.

This function returns -1 if the tmsid it gets passed isn't in the MFS directory "/StationTms". The backup and restore were both done with the same cable package, so I'd have expected them to both have the same sets of stations - but regardless, sometimes this function was getting to the error return of -1. I didn't look too hard to see why as it wasn't really the problem.

The problem is that set_station_logo doesn't test the value it gets back from station_tms_to_fsid, and so it falls over when it gets the "unknown" return. Ideally it should just puts stderr and skip that station.

I changed my version of backup-support to do this and then the next problem I hit was that backup had generated a call to set_station_logo with only one parameter. In fact, it generated a few lines like this:

set_station_logo 31611

as opposed to the more usual:

set_station_logo 21599 65539

This stops the script because set_station_logo is expecting two parameters.

I had a look and for me, TMS 31611 is "Living (Reduced)". It has no LogoIndex value associated with it, which is presumably why it's missing from the backup script output.

The backup script needs to test the logo index to see if it's empty and if it is, don't write the line out.

I've not had a look at the season passes and wishlist restores yet but I'll look soon. It looks like most of the errors are just missing checks for error conditions. If you went through and put those in everywhere then you'll probably cover most of the problems that would break the scripts.

I've attached (I hope!) the changes I made to backup-settings and backup - these just set the season pass directory based on the version number (nicked from TivoWeb), skip stations with no LogoIndex when generating the output from backup and skip missing stations when restoring the backup. I'm a TCL newbie, I usually write C++, so please feel free to fix anything rubbish. I've marked the changes with # agw comments. There really aren't a lot of them.

angra · 02-24-2003, 03:19 PM

I'm glad people are finding and correcting problems, and posting fixes. I've made a reference to your modified release in the top post so people will hopefully grab that.

mrtickle · 02-25-2003, 06:52 AM

Yep! Many thanks for this bit too:

if {$version3} {
set seasonpassdir "/SeasonPass/User"
} else {
set seasonpassdir "/SeasonPass"
}

havanahjoe · 02-27-2003, 01:35 AM

I did a backup of season passes and wishlists on a DirecTivo running version 3.1. I am now trying to restore this info on the same Tivo which is now running version 2.5.1. I got some errors at first, about the logo restoration, but since I don't need this I just erased all the logo entries on my backup file.
I ran the restore again and it starts working but it then gives me the following error:
object not found (errNmNameNotFound)

while executing
"mfs find $guideindexdir/Tms.key"
(procedure "tms_to_fsid" line 4)
invoked from within
"tms_to_fsid $tmsid"
("uplevel" body line 2)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set fsid [tms_to_fsid $tmsid]
} "
(procedure "rebuild_series" line 10)
invoked from within
"rebuild_series {SH080955 Version:63 ServerId:ATSH080955 ServerVersion:0 {Genre:
375 384 111 116 105 124} ThumbData:268633087 {Title:{The X-Files}} TmsI..."
(procedure "restore_seasonpasses" line 12)
invoked from within
"restore_seasonpasses"
(file ".//restore" line 27)

I think the problem is that it can't find the entry for this show in the guide... Is that it? It's very possible since I just reinstalled everything a few hours ago so the guide won't be completely populated for a few days.

Does the script need to find the shows in the guide?

havanahjoe · 02-27-2003, 01:42 AM

I just took a closer look at this and the problem is getting the station ID. I guess that 3.1 handles these in a different way, and now with 2.5 it can't find them. If this is the problem, can anything be done to fix it?

Another thing.. Does the script have to finish without errors for the changes to take effect? I have some wishlists that are spawned out by the script before season passes but they don't appear on the season pass manager.

agw · 02-27-2003, 08:03 AM

Quote:

Originally posted by havanahjoe
I did a backup of season passes and wishlists on a DirecTivo running version 3.1. I am now trying to restore this info on the same Tivo which is now running version 2.5.1.

<snip>

I ran the restore again and it starts working but it then gives me the following error:
object not found (errNmNameNotFound)

while executing
"mfs find $guideindexdir/Tms.key"

I've had a look at the script and it hard-codes guideindexdir to GuideIndexV2. Looking at TiVoWeb's index.itcl it appears that this is another thing that changed between V2 and V3 - on V2, which you're trying to restore to, it's /GuideIndex.

I've attached a modified backup-settings.itcl that sets guideindexdir to the right value. It's untested! But it should certainly fix the problem of going for the wrong directory.

As for whether the script has to finish without errors before changes take effect - I would expect that all changes up to the point where the error breaks the script will take effect. With the wishlists it just creates themes for them, so they should be there... but because you wouldn't have been able to add any season passes the associated season pass wouldn't have been created. I would have *expected* (but I'm probably wrong!) to see the wishlists in the "Search by wishlist" menu on the TiVo.

havanahjoe · 02-27-2003, 11:02 AM

Ok, that worked, I don't get that error anymore. But now I get this:

rebuilding series: SH080955 Version:63 ServerId:ATSH080955 ServerVersion:0 {Genr
e:375 384 111 116 105 124} ThumbData:268633087 {Title:{The X-Files}} TmsId:SH080
955
found prebuilt 7681
or:SeasonPass Version:7 MaxRecordings:5 {Series:@$fsid} {Station:@$station_id} P
riority:0 ShowStatus:0
doing it
minival=$fsid
id=7681
doing it
minival=$station_id
id=4004
invalid attribute: ShowStatus
while executing
"dbobj $new_obj set $type $subval"
("foreach" body line 3)
invoked from within
"foreach subval $val {
if {$index==0} {
dbobj $new_obj set $type $subval
} else {
dbobj $new_obj add $type $subval ..."
("uplevel" body line 21)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set new_obj [db $db create $type]
for {set i 1} {$i<[llength $object]} {incr i} {
set element [lindex $object $i]
..."
(procedure "object_rebuild" line 5)
invoked from within
"object_rebuild {SeasonPass Version:7 MaxRecordings:5 {Series:@$fsid} {Station:@
$station_id} Priority:0 ShowStatus:0}"
(procedure "restore_seasonpasses" line 15)
invoked from within
"restore_seasonpasses"
(file "./restore" line 27)

agw · 02-27-2003, 09:46 PM

Referring back to the ever-handy TiVoWeb scripts it appears that the ShowStatus field in the season pass record is version 3's new name for the FirstRun field.

I've grepped for "version3" in the TiVoWeb stuff and I think you'll probably be OK with the entry you're trying to insert, as long as you or the script just changes the text "ShowStatus" to "FirstRun". In this case it appears that only the name changed - the value assigned to the name is valid under both versions.

However I'm a little unsure as to how the TiVo will react to entries that have the same name between V2 and V3, but different valid sets of values. Going from V2 to V3 should be OK as you'd imagine (or rather, hope) that the valid V3 values would be supersets of the V2 values. However there's no reason to expect V2 software to accept all values that are valid under V3. The script could certainly be changed to normalise V3 values back to valid V2 ranges of values - but I've no idea what any of those ranges or values would be, or even if it's a problem at all. Some brave soul will have to try it and see what happens, if they haven't already.

Attached is another version of backup-support.itcl that has in place a mechanism for renaming these sub-types. I've only put in support for renaming between "FirstRun" and "ShowStatus" for the object type "SeasonPass" but it can easily be expanded to cover any other examples that crop up.

I've not tested this with a live script but I have tested the gobbet that does the name swapping, and for your example it did the job and didn't knacker any of the other names. I've put in a trace to screen to tell you that the name is being changed.

I would recommend that you clear out all season passes before running the restore script. I suspect that it will try to restore the "Priority" field in the season pass. This appears to be a number, zero-based, that determines the priority of the season pass. The TiVo might not be happy if more than one season pass ends up with the same priority. It might be an idea for the restore script to find the highest priority in the existing season passes on the machine and use that as a base for the new passes it inserts, and so ensure they remain unique - but it doesn't do this at the moment. Clearing out all existing season passes would probably be enough to avoid this - and like before, it might not even be a problem, the TiVo might be quite happy with it.

Good luck!

havanahjoe · 02-28-2003, 12:58 AM

We're making progress!! But there are still problems:

rebuilding series: SH080955 Version:63 ServerId:ATSH080955 ServerVersion:0 {Genr
e:375 384 111 116 105 124} ThumbData:268633087 {Title:{The X-Files}} TmsId:SH080
955
found prebuilt 7681
or:SeasonPass Version:7 MaxRecordings:5 {Series:@$fsid} {Station:@$station_id} P
riority:0 ShowStatus:0
doing it
minival=$fsid
id=7681
doing it
minival=$station_id
id=4004
Renaming sub-object ID ShowStatus for SeasonPass to FirstRun
The value assigned to this ID - 0 - remains unchanged...
rebuilding series: SH080955 Version:63 ServerId:ATSH080955 ServerVersion:0 {Genr
e:375 384 111 116 105 124} ThumbData:268633087 {Title:{The X-Files}} TmsId:SH080
955
found prebuilt 7681
or:SeasonPass Version:6 MaxRecordings:5 {Series:@$fsid} {Station:@$station_id} P
riority:1 ShowStatus:0
doing it
minival=$fsid
id=7681
doing it
minival=$station_id
id=4006
Renaming sub-object ID ShowStatus for SeasonPass to FirstRun
The value assigned to this ID - 0 - remains unchanged...
rebuilding series: SH229827 Version:59 ServerId:ATSH229827 ServerVersion:0 {Genr
e:369 371 105 124} ThumbData:268633087 {Title:{South Park}} TmsId:SH229827
found prebuilt 13274
or:SeasonPass Version:9 MaxRecordings:5 Priority:2 {Series:@$fsid} {Station:@$st
ation_id} ShowStatus:1
doing it
minival=$fsid
id=13274
doing it
minival=$station_id
id=4057
Renaming sub-object ID ShowStatus for SeasonPass to FirstRun
The value assigned to this ID - 1 - remains unchanged...
can't read "newids": no such variable
while executing
"lindex $newids [lsearch $oldids 38442]"
(procedure "restore_seasonpasses" line 27)
invoked from within
"restore_seasonpasses"
(file "./restore" line 27)

I deleted all the season passes I had already created, about 10, and ran the script but got this error after a while. It created 3 season passes and stopped. Sorry about all the problems..

agw · 02-28-2003, 07:05 AM

No worries about the problems - I'm impressed with your peserverance

The restore script runs the "restore_wishlists" function first and that runs through all of your wishlists, creating new identifiers for each of them and building up two global arrays, once called oldids that contains the original ID numbers for these wishlists and the other called newids that contains the ID number as it is now.

The line that is going over is where the restore_seasonpasses function, again in the script generated by backup, is searching oldids to find the element number that contains the old wishlist ID number and then getting the value for this from the newids array.

These arrays are global and the restore_wishlists function is obviously meant to run before restore_seasonpasses so that it can fill these arrays in. Have you changed the output from the backup script to empty out the restore_wishlists function, or modified restore to not call restore_wishlists before calling restore_seasonpasses?

I think the function to rebuild the wishlists will always try to create them, so it might be an idea to clean out the wishlists from the "Search by wishlist" menu before running the script.

If you haven't changed the restore script or the output from the backup file then it might be an idea if you attached the output from the backup script to a message so that I can see what's going on with your version of it.

angra · 02-28-2003, 09:05 AM

agw's analysis is right on.

if it can't find the "newids" variable it sounds like you're trying to run just the season pass restores. This is an intuitive thing to do, but won't work. Season Passes that are actually auto-record wishlists contain what amounts to a symbolic link inthe database object to the wishlist databse object. In order to restore these, the script has to be able to find the w/l object. The first lines in the restore script create the translation table variables (newid, etc), so if they don't exist at all, I suspect that's the trouble.

I considered putting in an option to clear out existing data before doing the restores (which would alleviate the temptation to run just one portion), but I was pretty convinced I would have accidentally destroyed all my settings during testing several times.

Thanks again to AGW for taking the ball and running with this one, and good luck with getting everything up & running havanahjoe.

havanahjoe · 02-28-2003, 10:35 AM

That could be the problem, I actually removed everything from the wishlist function in my output from the backup, because it had already created them. I'll use the original file, but this reminds me that if I use the restore file without any modifications, I get two different errors. The first one is caused by this line:

TmkLogger: <134>Feb 26 21:24:05 tcl[261]: Tcl created pool of 1458176 bytes

that was stored when the backup script ran. So I just erased it.

But then I'm getting problems with the logo Id's:

no value given for parameter "logoindex" to "set_station_logo"
while executing
"set_station_logo 12444 "
(procedure "restore_logos" line 3)
invoked from within
"restore_logos"
(file "./restore" line 25)

I got this errror the very first time I tried restoring the backup, so I removed all the entries in the restore logos function and that's when I started finding the other problems you guys have already helped me with.

The logos are not really something that I'm worried about, but I'm posting this error in case someone does care about them.

I'll remove the logos and try the script again and post my results.

havanahjoe · 02-28-2003, 10:41 AM

Well I got farther down the restore but I got another errror.

The script created all my wishlists, 6 season passes, 1 auto-recording wishlist and stopped with this:

rebuilding series: SH524467 Version:43 ServerId:ATSH524467 ServerVersion:0 {Genr
e:375 105 124} ThumbData:268566912 {Title:{John Doe}} TmsId:SH524467
can't read "db": no such variable
while executing
"db $db create Series"
("uplevel" body line 2)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set new_obj [db $db create Series]
for {set i 1} {$i<[llength $object]} {incr i} {
set element [lindex $object $i]
..."
(procedure "rebuild_series" line 23)
invoked from within
"rebuild_series {SH524467 Version:43 ServerId:ATSH524467 ServerVersion:0 {Genre:
375 105 124} ThumbData:268566912 {Title:{John Doe}} TmsId:SH524467}"
(procedure "restore_seasonpasses" line 31)
invoked from within
"restore_seasonpasses"
(file "./restore" line 27)

I'll attach my original backup file in case it helps.

Thanks again guys!